Tone Reproduction and Color Appearance Modeling: Two Sides of the Same Coin?

Color appearance models and tone reproduction algorithms are currently solving different problems. These classes of algorithms are also developed and used in different communities. However, they show remarkable functional similarities. Perhaps there is reason to think that they could in fact be one and the same thing. The advantages would be that we could achieve dynamic range reduction while taking human color vision into account. Vice-versa, we could predict the appearance of color over a large range of intensities. But how to overcome the differences, and how to construct an algorithm that could be both a tone reproduction model as well as a color appearance model? Introduction An imaging pipeline consists of processes to capture, store, transmit and display images and video. Traditional imaging pipelines are designed around the abilities of conventional capture and display devices, and therefore do not need dynamic range beyond what can be represented with a single byte per color channel. This situation is changing as image capture and in particular display technologies are maturing to include higher dynamic ranges [1]. High dynamic range imaging technologies produce and manipulate pixel data that conceptually consist of floating point numbers instead of 8-bit integer formats [2]. The benefit is clear: capturing data at full fidelity will lead to better imagery, even if the display device is not capable of reproducing the full dynamic range. An example is shown in Figure 1, where a single 8-bit exposure of a scene is compared with a high dynamic range (HDR) capture of the same scene. The resulting high dynamic range image was tonemapped to fit the reproduction range of paper. Note that the exposure on the left has both underand over-exposed areas. This is not uncommon and therefore a good example of the utility of high dynamic range imaging technologies. While representing pixels as floating point numbers rather than bytes may seem a minor change, there are many perceptual as well as technological aspects that require a reassessment. On the technological side, there are still many challenges. Perhaps the main one is that HDR image and video capture devices generate an enormous amount of data that would have to be managed. Standard compression algorithms are not directly amenable to HDR data [4, 5, 6, 7], with the implication that broadcast standards have yet to emerge. Second, HDR movie cameras are only just becoming available, including the Red Epic1 and the camera by Contrast Optical Engineering [8]. Third, it is not entirely clear how much dynamic range 1http://www.red.com/ Figure 1. This scene was captured with a single exposure (left) and high dynamic range imaging technologies (right). The image on the right was tonemapped for display/print using the photographic tone reproduction operator [3]. Photograph courtesy of Tania Pouli. should be captured. While the range of illumination between starlight and bright sunlight over which the human visual system can adapt is around 10 orders of magnitude [9], it seems overkill to try and capture this full range at all times. The human visual system is able to simultaneously perceive around 4 orders of magnitude of illumination under a specific laboratory set-up [10], although in practice this number may be a bit higher. It would probably be good practice to design imaging pipelines around this number. If it is assumed that HDR imagery and video will be captured with such a dynamic range, then displays should match this capability as well. Currently, only very few displays currently come even close, the Dolby prototype displays [1] and their commercial derivatives by SIM22 being the exception. Print technology is inherently incapable of reaching such dynamic range due to its reflective nature. Nonetheless, it may be foreseen that display devices will soon exhibit a greater variety in dynamic range than currently available. Whether low dynamic range legacy content or high dynamic range data is sent to a display, it will need to be mapped into a format that can be handled by that given display. In particular, it will need to be tonemapped to fit the dynamic range of the display device, and should take into consideration the state of adaptation of the observers. In recent years, much progress has been achieved in the design of algorithms that map high dynamic range images to low dynamic range display devices [2, 6]. Moreover, these algorithms have been subjected to psychophysical evaluation such as preference ratings [11, 12, 13] and similarity ratings [14, 13, 15, 16]. Although several tone reproduction operators are capable at compressing dynamic range, in this paper we argue that one weakness that persists is the lack of sensible color management. In particular, it is well-known that there exist luminanceinduced appearance phenomena such as the Hunt and Stevens ef2http://www.sim2.com/ fects, the Helmholz-Kohrausch effect and the Bezold-Brücke hue shift [17, 18] which indicate that there is a complex relationship between the perception of color and the luminance level at which colors are perceived. Currently, these effects are not generally taken into consideration in tone reproduction operators, leading to images that generally look either too vivid or too dull, and are certainly unsuitable for accurate color reproduction. On the other hand, color appearance modelling is an active area of research that has led to several models that predict the perception of color under different illumination conditions [17]. With the tristimulus values of a patch of color given, as well as a description of the environment in which it is observed, such models predict the perception of color in terms of appearance correlates, which include lightness, brightness, hue, saturation, colorfulness and chroma [19, 17, 18]. Few color appearance models are designed with high dynamic range imaging in mind, although notable exceptions exist [20, 21, 22, 23]. In particular, the models proposed by Kim et al. [22] are based on a psychophysical dataset that spans a much higher dynamic range than the psychophysical dataset that lies at the heart of most color appearance models [24]. The purpose of this paper is to argue that although tone reproduction and color appearance modelling may be addressing different problems, their aims partially overlap. Moreover, their functional similarity is unmistakable, albeit also with significant differences. This is especially the case for tone reproduction operators that model aspects of human vision. This paper catalogs the similarities and differences in order to show where the opportunities lie to construct a combined tone reproduction and color appearance model that could serve as the basis for predictive color management under a wide range of illumination conditions. It is thought that such an algorithm would benefit both fields of high dynamic range imaging as well as color imaging. To this end, the remainder of the paper begins by briefly describing the aforementioned luminance-induced appearance phenomena. Then, the structure of tone reproduction operators is outlined, insofar based on neurophysiology. These models are functionally closest to color appearance models, which are discussed next. A discussion of attempts to bring tone reproduction and color appearance modelling closer together then precedes the conclusions. Luminance Induced Appearance Phenomena The overall amount of light under which colors are observed may change the appearance of these colors. For instance, on a bright sunny day colors tend to appear more colorful than on an overcast day [18]. Several different observations have been made that relate to the relationship between illumination and color appearance. First, the Hunt effect states that as the luminance of a given color increases, so does its perceived colorfulness [25]. Further, perceived brightness contrast also changes with luminance, which is known as the Stevens effect [26]. Brightness itself is not only a function of luminance, but also depends on the saturation of the stimulus. This is described by the Helmholtz-Kohlrausch effect, although this effect depends on hue angle as well [27]. Finally, the perception of the hue of monochromatic light sources depends on luminance level, which is described by the Bezold-Brücke hue Figure 2. The image on the left was tonemapped with the photographic operator [3], which compresses the luminance channel of the Y xy color space. It therefore does not take luminance induced appearance phenomena into account. The image on the right was tonemapped using the color appearance model by Kim et al. [22]. cβ+kβE * τE τR 1/β / τX X nx